Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization

摘要

In practice, the parameters of control policies are often tuned manually.This is time-consuming and frustrating. Reinforcement learning is a promisingalternative that aims to automate this process, yet often requires too manyexperiments to be practical. In this paper, we propose a solution to thisproblem by exploiting prior knowledge from simulations, which are readilyavailable for most robotic platforms. Specifically, we extend Entropy Search, aBayesian optimization algorithm that maximizes information gain from eachexperiment, to the case of multiple information sources. The result is aprincipled way to automatically combine cheap, but inaccurate information fromsimulations with expensive and accurate physical experiments in acost-effective manner. We apply the resulting method to a cart-pole system,which confirms that the algorithm can find good control policies with fewerexperiments than standard Bayesian optimization on the physical system only.